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ABSTRACT 

Determining  the  proper  emphasis  of  curriculum  contents 
as  well  as  judging  the  value  or  worth  of  training  programs 
has  become  an  important  problem.   The  purpose  of  this  paper 
is  to  demonstrate  the  usefulness  of  task  analysis  in 
measuring  the  effectiveness  of  training  courses  based  on 
the  extent  to  which  curriculum  contents  are  job  oriented. 
In  this  regard,  parametric  and  nonparametric  statistical 
procedures  are  discussed  as  well  as  a  matrix  method  of 
evaluation.   A  general  methodology  to  include  the  opera- 
tional significance  of  the  data  is  also  included. 

Additionally,  the  results  of  a  small  scale  experiment 
are  presented.   This  experiment  was  conducted  to  determine 
the  most  valid  questionnaire  associated  data  collection 
method. 
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I.   INTRODUCTION 

The  increasing  technology  of  military  developments  has 
been  surpassing  the  military  services'  ability  to  supply 
competently  trained  manpower  for  meeting  the  knowledge  and 
technical  skill  requirements  for  particular  military  jobs. 
While  the  adequacy  of  military  training  is  dependent  on  the 
existence  of  effective  procedures  for  developing  and  main- 
taining the  appropriate  instructional  content,  a  means  of 
judging  the  value  or  the  worth  of  the  training  program  has 
also  become  important.   It  is,  however,  quite  misleading  to 
associate  any  training  model  or  training  curriculum  with  a 
strict  effective-ineffective  judgement,  since  training 
programs  may  be  considered  adequate  under  certain  circum- 
stances and  inadequate  under  others . 

In  an  attempt  to  achieve  a  high  positive  correlation 
between  job  proficiency  levels  and  the  associated  training 
costs,  all  of  the  military  services  are  striving  to  estab- 
lish instructional  systems  around  a  solid  foundation  of  job 
data  which  portrays  valid  instructional  objectives.   Since 
job  data  refers  to  a  statement  of  the  work  that  an  individual 
actually  performs  on  the  job,  the  ideal  source  for  the 
collection  of  this  data  is  the  incumbent  performing  in  his 
assigned  billet.   The  technique  of  identifying  and  collect- 
ing these  essential  elements  of  the  job  is  called  task 
analysis.   Basically,  by  utilizing  a  carefully  structured 


questionnaire  containing  a  composite  list  of  the  probable 
tasks  included  in  a  job,  as  well  as  a  rating  scale  for 
classifying  tasks  as  to  their  degree  of  necessity,  the 
extent  to  which  certain  elements  of  the  job  constitute  an 
individual's  performance  can  be  precisely  identified.   Once 
a  precise  identification  of  the  emphasis  devoted  to  specific 
duties  on  the  job  is  achieved,  training  objectives  can  be 
adjusted  in  order  to  bring  training  emphasis  more  on  target. 
Task  analysis,  therefore,  is  useful  in  determining  what 
training  should  be  given  and  when  it  should  be  given  in 
order  to  be  timely. 

Prior  to  the  introduction  of  task  analysis  procedures, 
course  contents  were  primarily  constructed  to  include  those 
areas  of  instruction  which  were  thought  to  be  required  for 
adequate  on  the  job  performance.   Additionally,  no  quanti- 
tative  methods  were  available  for  evaluating  training 
programs  within  the  military  services.   With  the  advent  of 
task  analysis  and  the  subsequent  numerical  taxonomy  of  job 
elements,  sufficient  data  has  become  available  for  the 
utilization  of  statistical  techniques  in  analyzing  the 
effectiveness  of  military  training. 

This  paper  is  based  on  the  continued  interest  of  the 
United  States  Marine  Corps  in  the  area  of  task  analysis  and 
job  data  utilization,  as  well  as  the  recently  initiated 
program  of  military  occupational  studies.   In  particular, 
the  primary  objective  of  the  paper  is  to  demonstrate  the 
usefulness  of  task  analysis  in  the  evaluation  of  military 
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training  programs.   More  specifically,  a  task  analysis 
performed  by  the  investigator  is  employed  as  a  data  base 
for  illustrating  some  of  the  appropriate  statistical  tech- 
niques available  for  evaluating  the  effectiveness  of  three 
related  Marine  Corps  training  courses.   Additionally,  an 
approach  toward  determining  the  operational  as  well  as  the 
statistical  significance  of  the  effectiveness  of  these  three 
courses  is  presented. 

As  a  secondary  objective,  validation  techniques  and 
statistical  comparisons  are  offered  for  three  task  analysis 
data  collection  methods  applicable  to  the  job  inventory 
questionnaire.   In  this  regard,  the  results  of  a  small  scale 
experiment  designed  to  determine  the  most  valid  questionnaire 
associated  data  collection  method   are  presented. 

It  is  hoped  that  this  research  will  allow  quantitative 
as  well  as  qualitative  training  curriculum  analysis  to  become 
a  more  standard  and  definitive  process  within  the  Marine 
Corps . 

A.   SCOPE  AND  LIMITATIONS 

One  formal  Marine  Corps  training  program  conducted  at 
the  Marine  Corps  Service  Support  Schools,  Marine  Corps  Base, 
Camp  Lejeune,  North  Carolina,  is  used  to  demonstrate  the 
usefulness  of  task  analysis  in  the  evaluation  of  military 
training.   The  training  program  is  subdivided  into  three 
separate  courses  designed  to  allow  trainees  to  advance  from 
one  course  to  the  next  as  they  advance  in  rank.   Basically, 
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the  program  is  structured  to  include  a  basic,  an  interme- 
diate, and  an  advanced  level  course,  and  is  devoted  to 
training  Marines  from  the  entry  level  rank  through  E-7  in 
the  area  of  supply  administration.   A  description  of  the 
subject  curricula  as  well  as  the  rank  requirements  for  each 
is  presented  in  Appendix  A. 

The  data  base  to  be  used  for  illustrating  the  analysis 
techniques  applicable  to  training  evaluation  was  collected 
from  a  representative  sample  of  Marines  serving  with  MOS 
(30*11)  supply  administrative  man  manual  and  MOS  (30^2)  supply 
administrative  man  mechanized.   Since  the  basic  training 
course  is  designed  to  prepare  Marines  for  filling  manual 
supply  billets,  only  Marines  with  MOS  (3041)  were  used  to 
provide  data  for  the  evaluation  of  this  particular  course. 
However,  the  intermediate  and  advanced  level  courses  are 
devoted  to  preparing  Marines  for  both  manual  and  mechanized 
supply  positions;  therefore,  Marines  with  MOS  (3041)  and 
MOS  (3042)  were  employed  to  provide  data  for  the  evaluation 
of  these  two  courses.   The  data  has  been  aligned  into  three 
distinct  categories  representative  of  the  Marine's  most 
recent  course  completion.   This  arrangement  provides  data  in 
an  easily  accessible  manner  for  evaluating  the  individual 
courses  as  well  as  the  overall  training  program. 

The  human  variables  contributing  to  successful  job  per- 
formance can  be  categorized  into  two  specific  areas  of 
human  factors  oriented  elements.   One  of  these  categoriza- 
tions includes  the  actual  units  of  work  that  form  a 
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significant  part  of  a  job  and  are  referred  to  as  performance 
tasks  or  task  elements.   The  other  categorization  includes 
the  worker  characteristics  required  for  satisfactory  job 
performance.   These  characteristics  are  not  related  to 
specific  duties  but  include  such  factors  as  physical  strength 
and  mental  requirements.   A  job  inventory  questionnaire  was 
selected  as  the  device  for  collecting  job  data  and  was 
designed  to  include  these  two  categorizations  of  human 
factors  oriented  elements. 

The  performance  task  portion  of  the  questionnaire  con- 
sists of  actual  task  elements  which  represent  the  essential 
part  of  the  research.   Data  obtained  from  responses  to  these 
selected  job  description  statements  is  used  to  demonstrate 
techniques  for  determining  whether  the  skills  needed  for 
satisfactory  performance  in  an  administrative  supply  billet 
are  consistent  with  the  course  contents  of  the  associated 
formal  training  curriculum.   The  performance  task  portion 
of  the  questionnaire  developed  by  the  investigator  is 
illustrated  in  Appendix  B. 

Although  the  primary  emphasis  of  this  paper  is  devoted 
to  the  analysis  of  actual  performance  task  data,  the  worker 
characteristic  segment  of  the  questionnaire  is  representative 
of  an  attempt  to  quantify  the  human  characteristics  required 
for  work  in  the  supply  field.   This  portion  of  the  question- 
naire was  developed  by  Professor  Gary  K.  Poock,  a  member  of 
the  Operations  Analysis  faculty  at  the  Naval  Postgraduate 
School  and  is  displayed  in  Appendix  C.   Obviously,  the  data 
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obtained  from  the  worker  characteristic  section  of  the 
questionnaire  does  not  lend  itself  to  training  curriculum 
evaluation  but  was  useful  in  providing  an  additional  data 
base  for  the  validation  and  comparison  of  different  data 
collection  methods.   The  information  obtained  from  this 
particular  portion  of  the  questionnaire  would  be  most 
appropriately  applied  to  an  analysis  of  MOS  assignment 
procedures . 

B.   ASSUMPTIONS 

The  basis  for  the  quantitative  evaluation  of  training 
effectiveness  presented  in  this  paper  rests  on  the  assump- 
tion that  the  most  important  tasks  performed  by  an  individ- 
ual on  the  job  are  the  ones  which  occupy  the  major  portion 
of  his  time.   Accordingly,  these  elements  of  the  job  which 
occupy  the  most  time  are  the  ones  in  which  students  should 
receive  the  most  intensive  training.   Basically,  a  student 
should  be  trained  to  perform  those  tasks  which  will  be 
required  of  him  in  the  field.   Of  course,  the  more  specific 
the  purpose  of  the  training  program  in  terms  of  performance 
requirements  for  particular  military  jobs,  the  more  appro- 
priately can  the  effectiveness  of  the  program  be  determined 
by  measuring  the  emphasis  devoted  to  specific  tasks  by  its 
graduates.   The  training  objectives  of  many  military  schools 
are  established  to  prepare  graduates  for  specific  job 
functions,  while  others  are  purposely  constructed  so  that 
the  dependence  of  job  performance  upon  the  training  program 
is  quite  indirect. 
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More  specifically,  the  approach  to  be  taken  in  this 
paper  implies  a  direct  relationship  between  the  training 
curriculum  and  the  job.   This  approach  represents  the  first 
disadvantage  in  the  assumptions  presented  above  in  that 
this  direct  relationship  is  not  characteristic  of  all 
training  programs.   Curriculums  devoted  to  providing  general 
background  type  knowledge,  or  those  normally  associated  with 
higher  level  and  more  complex  positions,  often  intentionally 
avoid  a  close  relationship  with  specific  elements  of  the 
job. 

Another  disadvantage  associated  with  the  assumptions 
presented  above  is  related  to  the  degree  of  difficulty 
inherent  in  each  of  the  elements  of  the  job.   Skills  which 
are  not  easily  learned  should,  in  most  cases,  receive  more 
emphasis  in  the  course  content.   In  this  regard,  some 
elements  of  a  job,  although  performed  less  frequently  than 
others,  require  more  intensive  instruction  to  produce  the 
desired  proficiency  level  of  the  trainee.   To  eliminate 
this  restriction,  relationships  between  job  proficiency 
levels  and  training  emphasis  could  be  established  for  the 
essential  elements  of  the  training  program. 

As  an  example  of  such  a  relationship  consider  the 
hypothetical  curves  shown  in  Figure  1.   These  curves  repre- 
sent the  job  proficiency  level  versus  the  training  emphasis 
devoted  to  two  maintenance  oriented  tasks.   Since  rotating 
truck  tires  is  relatively  simple  and  learned  rapidly, 
maximum  proficiency  is  achieved  quite  quickly.   In  contrast, 
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(arbitrary  time  units) 


Figure  1.   Hypothetical  Representation  of  Job 
Proficiency  Versus  Training  Emphasis 
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conducting  vehicle  safety  inspections  is  a  complex  task 
which  is  learned  more  slowly.   Accordingly,  improvement  in 
proficiency  can  be  obtained  from  additional  training  time. 
As  an  example  of  this  relationship,  consider  a  training 
program  which  devotes  five  training  units  to  rotating 
truck  tires  and  five  training  units  to  conducting  vehicle 
safety  inspections.   Suppose  the  program  was  developed  on 
the  basis  of  a  survey  demonstrating  that  approximately  equal 
times  are  devoted  to  these  tasks  on  the  job.   Inspection  of 
the  graph  indicates  that  if  two  training  units  were  elimin- 
ated from  rotating  tires,  no  reduction  in  proficiency  would 
occur.   These  same  two  units  of  training  could  then  be 
applied  to  increase  the  proficiency  level  of  the  more  com- 
plex task  by  a  significant  amount  equal  to  (d).   Altering 
the  curriculum  in  this  way  certainly  represents  an  improve- 
ment in  the  overall  effectiveness  of  the  program. 

Additionally,  the  aspect  of  task  suitability  for  "on  the 
job  training"  emphasis  is  representative  of  still  another 
disadvantage  with  the  assumptions  employed  in  this  paper. 
As  an  example  of  this  aspect  of  training  to  the  attainment 
of  job  proficiency  levels,  consider  the  hypothetical  curves 
shown  in  Figure  2.   The  graph  suggests  that  two  independent 
tasks  possibly  occupying  equivalent  amounts  of  actual  per- 
formance time  are  not  equal  in  their  applicability  to 
training  on  the  job.   Task  number  1  is  relatively  simple  and 
indicates  the  achievement  of  a  high  proficiency  level  in  a 
very  short  time  period.   In  particular,  this  task  is 
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On  the  Job  Training  Emphasis 
(arbitrary  time  units) 


Figure  2.   Hypothetical  Representation  of  Job 
Proficiency  Versus  On  the  Job  Training 
Emphasis 
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definitely  suitable  for  on  the  job  training  and  a  likely 
candidate  for  elimination  from  the  formal  curriculum.   Task 
number  2  is  more  complex  and  does  not  appear  to  be  appli- 
cable to  training  techniques  of  this  type. 

Although  the  concepts  just  examined  have  substantial 
theoretical  value  and  appear  to  be  feasible  on  the  surface, 
they  represent  several  very  difficult  obstacles .   The 
inclusion  of  these  notions  would  require  a  variety  of  arbi- 
trary decisions  beyond  the  realm  of  this  paper.   It  is, 
however,  important  to  understand  that  deficiencies  exist  in 
the  subsequent  evaluation  procedures  if  the  previously 
mentioned  complex  issues  are  not  included.   Additionally, 
it  is  important  to  realize  that  accepting  the  basic  assump- 
tions will  produce  valid  training  program  evaluations 
provided  that  qualitative  considerations  are  applied  subse- 
quent to  the  quantitative  analysis. 

C .   METHODOLOGY 

The  evaluation  techniques  to  be  demonstrated  are  based 
on  statistical  comparisons  of  the  following  two  sets  of 
data: 

1.  The  relative  importance  of  the  tasks  involved  in 
the  job. 

2.  The  training  emphasis  devoted  to  the  tasks  involved 
in  the  job. 

The  first  set  of  data  was  obtained  by  utilizing  the  task 
inventory  questionnaire  in  conjunction  with  a  personal  inter- 
view and  actual  observation  of  the  incumbents'  performance 
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on  the  job.   Basically,  individuals  were  required  to  report 
the  extent  to  which  the  tasks  listed  on  the  questionnaire 
occupied  their  time.   In  keeping  with  the  basic  assumption 
of  this  paper,  those  tasks  which  occupied  the  most  time 
were  considered  to  be  the  most  important.   To  limit  the 
selection  of  responses  available  for  reporting  the  extent 
to  which  specific  tasks  occupied  their  time,  a  four  point 
selection  scale  was  used.   Incumbents  considered  each  task 
with  the  investigator  and  were  required  to  explain  their 
selected  response  from  the  four  point  scale.   It  was 
anticipated  that  the  required  explanation  would  insure  a 
complete  understanding  of  each  question,  while  at  the  same 
time  limit  the  workers  from  over  responding.   The  question- 
naire employed  and  the  associated  scoring  procedure  is 
illustrated  in  Appendix  C.   Scores  were  converted  to  per- 
centages to  provide  more  meaningful  data  In  nature  of  the 
comparisons  to  be  made.   A  computer  program  was  developed  to 
consolidate  and  display  the  data  in  an  orderly  manner.   The 
task  analysis  questionnaire  was  administered  to  95  enlisted 
Marines  ranking  from  E-2  through  E-8.   Once  collected,  the 
data  was  classified  in  accordance  with  the  Marines '  most 
recent  course  completion.   As  previously  mentioned  this 
classification  provided  data  in  an  easily  accessible  manner 
for  evaluating  the  three  courses  as  well  as  the  overall 
program.   The  results  of  this  classification  provided  29 
subjects  for  the  basic  course  evaluation,  50  subjects  for 
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the  intermediate  level  course  analysis,  and  16  subjects  for 
the  evaluation  of  the  advanced  course. 

The  second  set  of  data  was  obtained  from  theoretical 
proficiency  levels  specified  by  the  directors  of  the  train- 
ing program. 

Comparisons  of  these  two  sets  of  data  will  determine  if 
the  training  program  is  devoted  to  placing  emphasis  on  the 
appropriate  instructional  areas.   Statistical  techniques  are 
used  to  illustrate  some  of  the  available  procedures  for 
evaluating  the  three  courses  on  an  individual  basis.   The 
results  of  these  individual  course  evaluations  are  then 
extended  in  order  to  demonstrate  methods  for  determining  the 
effectiveness  of  the  overall  program. 

D.   TASK  ANALYSIS  DATA  COLLECTION  METHODS  EMPLOYED 

Although  a  variety  of  task  analysis  data  collection 
methods  exist,  those  employed  in  this  study  are  representa- 
tive of  the  procedures  applicable  to  the  job  inventory 
questionnaire.   The  questionnaire  permits  extensive  sampling 
of  military  occupational  specialties  in  a  relatively  short 
period  of  time.   Additionally,  the  questionnaire  oriented 
procedures  are  economically  administered  and  allow  large 
masses  of  data  to  be  reduced  for  computer  storage,  display 
and  analysis.   For  these  reasons  it  is  estimated  that  the 
data  collection  methods  employed,  although  far  from  exhaus- 
tive, are  representative  of  those  feasible  for  Marine  Corps 
efforts  in  this  area.   The  following  are  the  task  analysis 
job  inventory  procedures  utilized  in  this  paper: 
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1.  Personal  interview  and  actual  observation  of  an 
individual's  on  the  job  performance. 

2.  Distribution  of  the  task  analysis  inventory 
questionnaire  in  a  manner  not  permitting  observation  or 
interview  of  the  job  incumbent.   This  procedure  requires 
individual  completion  of  the  questionnaire  with  no  explan- 
ation of  the  task  statements. 

3.  Utilization  of  the  task  analysis  questionnaire  by 
officers  and  senior  staff  non-commissioned  officers  serving 
in  a  supervisory  capacity.   For  this  method  job  data  was 
obtained  by  supervisors  reporting  the  job  characteristics 
required  of  their  subordinates. 

Only  the  data  collected  using  the  questionnaire  in 
conjunction  with  a  personal  interview  and  observation  of  the 
job  incumbent  was  used  in  the  evaluation  procedures  demon- 
strated in  this  paper.   The  other  data  gathering  procedures 
were  useful  in  determining  the  most  valid  method  for  acquir- 
ing job  inventory  data. 
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II.   STATISTICAL  TESTING  IN  GENERAL 

The  purpose  of  a  training  course  evaluation  is  to 
provide  information  regarding  the  effectiveness  of  the  cur- 
riculum content.   Of  particular  interest  here  is  to  examine 
alternative  approaches  toward  determining  whether  or  not 
training  is  being  conducted  in  the  appropriate  instructional 
areas  and  if  students  are  of  the  proper  rank  in  order  for 
the  training  to  be  timely.   As  an  example  of  the  concept  of 
timely  training,  consider  a  motor  transport  mechanic  who 
at  some  time  in  his  career  may  be  required  to  employ  manage- 
ment techniques  in  the  operation  of  a  large  repair  facility. 
Since  these  management  techniques  will  probably  only  be 
required  of  senior  enlisted  men,  a  timely  training  program 
would  not  provide  instruction  in  this  area  for  the  entry 
level  or  even  the  intermediate  level  ranks.   What  seems  to 
be  needed  here  is  a  good  method  for  determining  the  effec- 
tiveness of  course  contents  while  at  the  same  time  providing 
a  foundation  capable  of  incorporating  the  ideas  mentioned 
above  into  an  evaluation  of  the  overall  program.   Possible 
approaches  toward  achieving  this  method  include  the  use  of 
nonparametric  as  well  as  parametric  statistical  techniques. 

A  parametric  statistical  test  is  a  test  in  which 
specific  assumptions  about  the  parameters  of  the  sampled 
population  are  made,  such  as  y,  =  y?  or  c-,  =   a~,  whereas 
a  nonparametric  test  makes  no  assumptions  about  the  value 
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of  the  parameters  in  the  sampled  population.   It  is  impor- 
tant to  distinguish  the  difference  in  the  assumptions  v;hich 
must  be  made  when  testing  parametrically  vice  nonpara- 
metrically.   The  basic  assumptions  of  the  parametric  test 
include  independence  of  observations,  underlying  normal 
distribution  of  the  sampled  populations,  and  homoscedacity 
of  the  population  variances.   The  assumptions  associated 
with  nonparametric  tests  Include  only  the  independence  of 
observations,  and  that  the  sampled  populations  be  continuous. 
When  judged  by  the  criterion  of  power  efficiency,  nonpara- 
metric tests  are  often  superior  to  their  parametric  counter 
parts  when  all  of  the  assumptions  required  of  the  parametric 
tests  can  not  be  met  [1] . 

In  keeping  with  the  primary  objective  of  this  paper, 
which  is  demonstrating  the  usefulness  of  task  analysis  In 
the  evaluation  of  military  training,  an  attempt  is  not  made 
to  determine  which  test  or  even  which  type  of  test  is  the 
best  for  such  an  analysis.   Rather  than  make  such  a  deter- 
mination, a  variety  of  applicable  evaluation  procedures  are 
presented.   In  addition  to  presenting  applicable  statistical 
evaluation  procedures,  the  concept  of  operational  effective- 
ness and  a  general  methodology  for  standardizing  military 
training  evaluations  are  examined. 
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III.   NONPARAMETRIC  EVALUATIONS  OF  TRAINING  EFFECTIVENESS 

A.  TESTS 

The  applicable  tests  selected  to  demonstrate  nonpara- 
metrlc  statistical  procedures  for  evaluating  the  effective- 
ness of  the  Marine  Corps  training  courses  are  the  Kolmogorov- 
Smirnov  Test  and  the  Spearman  Rank  Correlation  Coefficient. 
A  matrix  method  of  evaluation,  also  a  form  of  nonparametric 
testing,  will  be  examined  in  a  separate  chapter. 

B,  KOLMOGOROV-SMIRNOV  TEST 

Perhaps  the  most  huerestic  of  the  statistical  tests  is 
the  Kolmogorov-Smirnov  One  Sample  Test,  often  referred  to  as 
the  Smirnov  Maximum  Deviation  Test.   The  test  statistic  is 
the  maximum  deviation  between  two  empirical  distribution 
functions  [4],   Since  training  effectiveness  in  this  paper 
is  to  be  determined  by  evaluating  the  extent  to  which  course 
contents  are  job  oriented,  the  actual  proportions  of  job 
time  devoted  to  the  different  task  elements  are  used  as 
the  standard.   Therefore,  the  distribution  of  job  time  over 
the  task  elements  is  considered  the  theoretical  fixed  cumu- 
lative distribution.   Basically  the  data  in  each  category, 
i,e,,  basic,  intermediate,  and  advanced,  was  collected  using 
the  four  point  rating  scale  displayed  in  Appendix  C  and  was 
then  individually  averaged  over  every  task  statement  on  the 
questionnaire.   These  averages  were  then  converted  to  per- 
centages in  order  to  provide  the  average  proportion  of  time 
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devoted  to  each  task  element  by  the  most  recent  graduates 
of  each  of  the  three  Marine  Corps  training  courses.   More 
specifically,  a  fixed  theoretical  cumulative  distribution 
of  job  time  was  derived  for  each  of  the  three  categories  of 
actual  job  data. 

The  other  empirical  distribution  was  determined  from  the 
directors  of  the  training  program  and  represents  the  propor- 
tion of  time  devoted  to  the  task  elements  within  each  of  the 
three  courses.  So  at  this  stage  of  the  procedure,  a  theo- 
retical fixed  cumulative  distribution  of  actual  job  time  and 
a  cumulative  distribution  of  training  emphasis  were  available 
for  each  of  the  training  courses. 

To  compute  the  test  statistic,  list  the  values  of  the 
cumulative  distribution  functions  of  training  emphasis  and 
job  time  in  order  of  the  various  task  elements.   The  test 
statistic,  D,  is  max  I d .  I  .   Where: 

n       n 
d   =   £  J   -   Z  T      i  =  1,2,. ..n,. . .32, 
r    i=l  x    i=l  x 

and     J.  =  average  proportion  of  actual  job  time  devoted 
to  task  element  i. 

T.  =  proportion  of  training  time  devoted  to  task 
element  i. 

Testing  is  conducted  under  the  null  hypothesis,  H  :,  that 

the  two  distributions  are  the  same  and  the  alternative 

hypothesis,  H  • ,  that  the  two  distributions  are  not  the  same, 

It  is  important  to  note  that  the  Kolmogorov-Smirnov  Test 

used  in  the  manner  explained  above  does  not  meet  all  of  the 
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prerequisites  required  of  nonparametric  statistical  tests. 
The  data  is  cast  into  intervals  defined  by  each  of  the 
tasks  listed  on  the  questionnaire,  and  therefore,  cannot 
meet  the  continuity  assumption  required  for  this  nonpara- 
metric test.   Additionally,  the  results  of  the  Kolmogorov- 
Smirnov  Test  used  in  this  way  produce  a  somewhat  pessimistic 
appraisal  of  the  effectiveness  of  the  training  courses, 
since  the  value  of  D  will  be  at  least  as  great  as  the 
maximum  difference  between  the  job  time  versus  training 
emphasis  on  any  one  task  element.   The  test,  however,  has 
intuitive  appeal  in  that  it  considers  proportional  differ- 
ences on  distinct  task  elements  as  well  as  runs  of  task 
statements  which  display  discord  between  the  respective 
proportions . 

As  an  illustration  of  the  manner  in  which  this  test 
was  employed,  consider  the  data  associated  with  the  Basic 
Supply  Course.   An  explanation  of  the  nature  of  this  course 
is  presented  in  Appendix  A.   The  dat-a  points  of  the  two 

cumulative  distributions  as  well  as  the  d.fs  are  shown  in 

1 

Table  I.   Similar  tables  for  the  remaining  courses  are 
presented  in  Appendix  D.   Table  II  illustrates  a  general 
plot  of  the  two  cumulative  step  functions  associated  with 
the  Basic  Supply  Course. 
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TABLE  I 

CUMULATIVE  DISTRIBUTION  DATA  POINTS  IN  KOLNOGOROV- 
SMIRNOV  TEST  WITH  BASIC  COURSE  DATA 


Task  No. 

Training  (CDF) 

Job  (CDF) 

d. 

1 

1 

.02 

.0018 

.0182 

2 

.02 

.0036 

.0164 

3 

.02 

.0250 

.0050 

H 

.02 

.0250 

.0050 

5 

.09 

.1015 

.0115 

6 

.10 

.1086 

.0086 

7 

.28 

.1816 

.0984 

8 

.40 

.2688 

.1312 

9 

.51 

.2955 

.2145 

10 

.56 

.3346 

.2254* 

11 

.56 

.3666 

.1934 

12 

.56 

.3755 

.1845 

13 

.56 

.3986 

.1614 

14 

.56 

.4591 

.1009 

15 

.57 

.5036 

.0664 

16 

.60 

.5890 

.0110 

17 

.60 

.6566 

.0566 

18 

.60 

.6993 

.0OQ3 

19 

.61 

.7456 

.1356 

20 

.67 

.7634 

.0934 

21 

.68 

.7794 

.0994 

22 

.73 

.8257 

.0957 

23 

.92 

.8542 

.0658 

24 

.96 

.8791 

.0809 

25 

.96 

.8791 

.0809 

26 

.96 

.8846 

.0754 

27 

.99 

.9273 

.0627 

28 

1.00 

.9842 

.0158 

29 

1.00 

1.0000 

0 

30 

1.00 

1.0000 

0 

31 

1.00 

1.0000 

0 

32 

1.00 

1.0000 

0 

*D  =  .2254 
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TABLE  II 

CUMULATIVE  DISTRIBUTION  STEP  FUNCTIONS  IN  KOLMOGOROV- 
SMIRNOV  TEST  WITH  BASIC  COURSE  DATA 
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D  occurs  at  i  =  9  where 

d9  =  .56  -  .33^6  =  .225^. 

When  compared  with  the  critical  value  of  . 1613  it  is 
possible  to  reject  the  null  hypothesis  at  a  significance 
level  of  .01.   The  values  of  D  for  the  courses  under 
examination  are  shown  in  Table  III.   The  null  hypothesis 
when  applied  to  the  remaining  courses  may  also  be  rejected 
at  the  .01  significance  level. 

A  rejection  of  the  null  hypothesis  at  the  .01  signifi- 
cance level  for  all  three  courses  indicates  that  training 
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TABLE  III 
SUMMARY  OF  THE  KOLMOGOROV-SMIRNOV  TEST  RESULTS 

D 

Basic  Course  .225^ 

Administrative  Course  .1824 

Senior  Course  .2736 


emphasis  is  not  devoted  to  the  appropriate  instructional 
areas.   However,  more  important  than  the  statistical  sig- 
nificance determined  by  the  Kolmogorov-Smirnov  Test  is  the 
manner  in  which  the  test  presents  the  data.   The  examination 
of  the  two  cumulative  distributions  from  a  more  operational 
standpoint,  may  indicate  feasible  adjustments  for  bringing 
training  emphasis  more  on  target. 

As  an  example  of  some  of  the  comparisons  that  can  be 
made,  consider  the  illustration  of  the  two  cumulative  dis- 
tribution functions  presented  in  Table  II.   Tasks  nine 
through  fifteen  represent  a  trend  toward  overtraining.  The  over- 
emphasis devoted  to  training  on  these  tasks  may  be  more 
appropriately  applied  to  tasks  twenty  through  twenty  five, 
where  an  undertraining  trend  is  indicated.   Similar  compari- 
sons can  be  made  among  the  three  courses  in  order  to  locate 
possible  adjustments  for  improving  the  training  program  on 
an  overall  basis.   More  elaborate  comparisons  could  be  made 
for  evaluations  containing  a  much  greater  number  of  perform- 
ance tasks,  which  could  be  classified  and  sequenced  into 
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distinct  instructional  areas.   Nevertheless,  the  Kolmogorov- 
Smirnov  Test,  as  illustrated,  is  useful  In  providing  an 
indication  of  the  effectiveness  of  the  training  courses. 

C.   SPEARMAN  RANK  CORRELATION  COEFFICIENT 

Of  all  the  statistical  tests  based  on  ranks,  the  Spearman 
Rank  Correlation  Coefficient  is  perhaps  the  oldest  and  the 
most  widely  known.   The  test  statistic  r  is  a  measure  of  the 
degree  of  association  that  exists  between  two  groups  of 
data  [5]. 

To  compute  r  it  is  necessary  to  establish  two  separate 
categories  of  data  based  on  the  assignment  of  ranks  to  the 
tasks  listed  on  the  questionnaire.   The  first  ranking  is 
based  on  the  relative  importance  of  the  tasks  to  satisfac- 
tory performance  on  the  job  while  the  second  is  achieved 
from  the  emphasis  devoted  to  the  tasks  in  the  training  pro- 
gram.  The  task  determined  the  most  important  to  the  job  is 
assigned  a  rank  of  1  for  the  job  data  category  while  the 
task  receiving  the  most  intensive  training  emphasis  is 
assigned  a  rank  of  1  for  the  training  data  category.   The 
remaining  tasks  are  ranked  accordingly. 

The  test  statistic 

N 
6  I   d. 

r  =  1  - 


N3  -  N 


where 


d.  =  the  difference  between  the  two  ranks  assigned  to 
task  element  i. 
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N  =  the  number  of  task  elements. 

The  value  of  r  ranges  between  0  and  1.   The  highest 
possible  degree  of  association  is  1,  and  can  be  achieved 
if,  and  only  if,  each  task  element  receives  the  identical 
ranking  within  the  two  categories  of  data. 

If  the  proportion  of  ties  within  the  rankings  of  either 
category  is  large,  it  becomes  appropriate  to  adjust  the 
value  of  r  by  a  correction  factor.   The  correction  factor, 

n   t.3  -  t. 
T1  =   2    X     X  ,   j  =  X,Y 
J    i=l     12 

where : 

t.  =  the  number  of  observations  tied  at  rank  i. 

n   =  the  total  number  of  ranks  at  which  tied  obser- 
vations exist  within  a  given  category  of  data. 

The  variable  X  represents  the  training  data,  while  the 

variable  Y  represents  the  job  data.   With  the  correction 

factor  incorporated  one  can  use  the  formula: 

2    2     N 
X  +  Y   -   Ed. 

-    i=l  1 

r  =  — 

2yjx2    •  Y2 
where : 

2  _  N3  -N 
X       12"  "  TX 

2  _  N3  -N 

12   "   Y* 

The  rankings  required  for  the  computation  of  r  for  the  cur- 
ricula under  observation  are  listed  in  Appendix  E. 
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Additionally,  it  is  possible  to  test  the  significance 
of  the  value  of  r  or  r.   This  can  be  done  by  testing  the 
null  hypothesis,  Ho:  that  the  two  variables  under  observa- 
tion are  not  associated  against  the  alternative  hypothesis, 
H,:  that  there  is  a  measure  of  association  between  the  two 
variables.   The  number  of  task  elements  being  greater  than 
ten  determines  the  use  of  the  significance  test  for  large 
samples  [5].   The  significance  of  an  obtained  r  under  the 
null  hypothesis  may  be  tested  by: 


N  -2 

The  test  statistic,  t,  has  the  student's  t  distribution  with 

degrees  of  freedom  =  N-2,  for  large  values  of  N. 

Table  IV  illustrates  a  summary  of  the  results  obtained 

by  using  the  Spearman  Rank  Correlation  Coefficient. 

TABLE  IV 

SUMMARY  OF  SPEARMAN  RANK  CORRELATION  COEFFICIENT  COMPUTATIONAL 
RESULTS  OF  JOB  TIME  VERSUS  TRAINING  EMPHASIS 


Basic  Course  .557      .531      5. M6 

Administrative  Course       .282       .279      1.60 
Senior  Course  .366       .357      2.1^10 

Since  the  values  of  r  are  so  closely  related  to  r,  the  sig- 
nificance test  was  only  conducted  on  the  uncorrected  measures 
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of  association.   The  results  indicate  that  the  null  hypo- 
thesis can  be  rejected  in  the  following  instances: 

1.  Basic  Course  at  a  significance  level  of  .001; 

2.  Administrative  Course  at  a  significance  level  of  .20; 

3.  Senior  Course  at  a  significance  level  of  .05. 

A  rejection  of  the  null  hypothesis  indicates  that  a  measure 
of  association  does  exist  between  training  emphasis  and  job 
importance  for  the  tasks  listed  in  the  questionnaire. 
Therefore,  the  values  of  r  computed  for  the  three  training 
courses  are  statistically  significant  at  the  levels  shown 
above . 

In  addition,  qualitative  analysis  is  required  here  in 
order  to  determine  if  the  values  of  r  are  close  enough  to 
1  in  order  for  the  training  program  to  receive  an  effective 
rating.   The  establishment  of  a  standard  more  easily 
obtainable  than  a  correlation  coefficient  of  1  may  prove 
appropriate  in  order  for  this  test  to  produce  a  clear  eval- 
uation of  the  effectiveness  of  training.   Although  such  an 
extension  of  this  test  would  certainly  provide  more  meaning- 
ful results,  it  is  beyond  the  realm  of  this  paper  to  attempt 
even  an  approximation  of  this  standard  value. 

The  Kendall  Rank  Correlation  Coefficient  is  another 
measure  of  association  type  test  that  is  appropriate  here 
but  will  not  be  demonstrated  in  this  paper. 
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D .   SUMMARY 

This  concludes  a  presentation  of  possible  nonparametric 
statistical  tests  which  could  be  used  in  evaluating  the 
effectiveness  of  military  training.   The  results  of  the  two 
tests  conducted  in  this  chapter  appear  to  be  conflicting. 
The  Kolmogorov-Smirnov  Test  suggested  that  training  emphasis 
was  not  being  devoted  to  the  proper  instructional  areas 
while  the  Spearman  Rank  Correlation  Coefficient  indicated  a 
degree  of  association  between  training  emphasis  and  job 
importance  levels.   While  it  is  difficult  to  explain  the 
cause  of  these  differences,  it  is  certainly  safe  to  say 
that  the  assumptions  of  the  Spearman  Rank  Correlation 
Coefficient  are  most  closely  adhered  to  while  the  violation 
of  the  continuity  requirement  in  the  Kolmogorov-Smirnov 
Test  may  tend  to  discount  the  statistical  validity  of  its 
results . 
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IV.   A  MATRIX  METHOD  FOR  THE  EVALUATION  OF  TRAINING 
CURRICULUM  EFFECTIVENESS 

A.   INTRODUCTION 

The  matrix  method  of  training  curriculum  analysis  pre- 
sented here  has  been  adopted  from  a  study  conducted  by 
Arthur  I.  Siegel,  Douglas  G.  Schultz,  and  Phillip  Federman 
in  1961  for  the  Office  of  Naval  Research  [7] . 

This  correlation  oriented  procedure  enables  the  analyst 
to  determine  a  training  index,  an  undertraining  index,  and 
an  overtraining  index  for  each  curriculum  under  evaluation. 
The  numerical  value  of  each  index  will  be  a  number  greater 
than  or  equal  to  zero  and  less  than  or  equal  to  one.   The 
training  index  is  representative  of  the  overall  effective- 
ness of  the  program  while  the  overtraining  and  undertraining 
indexes  represent  the  direction  of  problem  areas  if  the 
curriculum  is  classified  as  ineffective.   A  training  index 
of  one  represents  the  highest  possible  positive  correlation 
between  adequate  job  performance  and  the  associated  task 
proficiency  levels  established  by  the  training  program. 
Basically,  an  effective  curriculum  content  should  produce 
a  training  index  close  to  one  while  the  undertraining 
and  overtraining  indexes  should  be  near  zero. 

Although  the  matrix  method  requires  some  value  judgement 
as  to  the  placement  of  tasks  in  the  matrix  cells,  its  basic 
advantage  is  the  relative  ease  with  which  calculations  can 
be  accomplished. 
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B.   METHOD 

The  procedure  is  based  on  the  categorization  of  tasks 
into  four  groups  labeled  "very  high,"  "high,"  "moderate," 
and  "low."   Separate  classifications  are  conducted  for  the 
training  level  associated  with  each  task  as  well  as  the 
relative  importance  of  each  task  to  performance  on  the  job. 
Subsequent  to  these  classifications  each  task  is  then  sorted 
into  one  of  the  sixteen  cells  in  the  matrix  shown  in  Figure 
3.   Cell  A,  for  example,  contains  tasks  which  are  very 
highly  important  and  for  which  trainees  have  been  trained 
to  a  very  high  proficiency  level.   Cell  H  contains  tasks 
which  are  greatly  important  but  received  little  training 
emphasis.   By  assigning  weights  to  the  cells  of  the  matrix, 
the  three  indexes  can  be  calculated  as  follows: 

1.   Training  Index  (TI):   If  all  tasks  were  instructed 
to  a  level  equivalent  to  that  required  for  job  performance, 
one  would  anticipate  all  of  the  tasks  to  locate  in  cells 
A,F,K,  and  P.   These  cells  are  assigned  a  weight  of  three. 
Cells  B,G,L,E,J,  and  0  are  assigned  the  weight  of  two,  cells 
C,H,I,  and  N  are  assigned  the  weight  of  one,  while  cells  D 
and  M  are  assigned  the  weight  of  zero.   Figure  4  illustrates 
the  assignment  of  weights  to  the  cells  of  the  matrix. 

The   urriculum  Training  Index  is  determined  by  the  sum 
of  all  the  weights  associated  with  the  tasks  of  a  particular 
curriculum  divided  by  three  times  the  number  of  tasks  that 
are  being  evaluated.   The  mathematical  expression  is  as 
follows : 
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Figure  3.   Matrix  Scheme  for  Task  Classification  by 
Frequency  of  Performance  and  Training  Level 
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Figure  k.      Weights  of  Matrix  Cells  for 
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N 

I      W. 

ti  =  £=i_i 

3N 

where : 

W.  =  the  weight  assigned  to  the  cell  in  which  task 
i  has  been  classified. 

N   =  the  number  of  tasks  being  classified. 

2.  Overtraining  Index  (01):   The  realization  of  a 
relatively  low  training  index  indicates  the  appropriateness 
of  measuring  the  direction  of  ineffectiveness  in  the  train- 
ing program.   Basically,  this  means  a  determination  of  the 
extent  to  which  individuals  are  being  overtrained  or  under- 
trained. 

A  measure  of  the  tendency  toward  overtraining  can  be 
determined  by  the  assignment  of  weights  to  the  matrix  cells 
as  shown  in  Figure  5.   Only  the  cells  that  indicate  over- 
training have  been  assigned  a  weight.   Computationally,  the 
overtraining  is  as  follows: 

N 

I      W 

oi  =  £=2 — 

3N 

Ideally,  an  effective  training  curriculum  will  produce  an 
overtraining  index  of  zero. 

3.  Undertraining  Index  (UI):   This  index  represents  a 
measure  of  the  tendency  toward  undertraining  and  is  deter- 
mined in  accordance  with  the  principles  shown  above.   Matrix 
cells  are  weighted  as  illustrated  in  Figure  6.   Only  the 
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cells  indicating  undertraining  have  been  assigned  a  weight. 

Computationally,  the  Undertraining  Index  is  as  follows: 

N 

I      W. 

UI  =  1=1 


3N 

The  three  indexes  for  each  training  course,  taken  toge- 
ther, provide  a  useful  summary  of  training  effectiveness  in 
each  instructional  area.   While  the  indexes  themselves 
provide  a  valuable  indication  of  the  effectiveness  of  train- 
ing, the  task  classification  procedure  is  more  specific  in 
pointing  out  where  increases  and  decreases  in  training 
emphasis  are  required. 

C.   ANALYSIS  AND  APPLICABILITY 

As  an  application  of  the  matrix  method  of  evaluation, 
the  three  indexes  were  calculated  for  each  of  the  courses 
under  study.   The  results  are  shown  in  Table  V.   The 
training  level  classification  for  each  task  was  provided  by 

TABLE  V 
RESULTS  OF  THE  MATRIX  METHOD  OF  EVALUATION 


n  Training    Overtraining    Undertraining 

Course         _  ,    °       x  ,  t~^~~ 

Index         Index  Index 


Basic 

.802 

Administrative 

.813 

Senior 

.823 

.073  .125 

.09^  .09^ 

.125  -052 
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the  directors  of  the  training  program,  while  the  frequency 
of  performance  level  was  determined  in  accordance  with  the 
criteria  displayed  in  Table  VI. 

TABLE  VI 
TASK  FREQUENCY  OF  PERFORMANCE  CLASSIFICATION  CRITERIA 


Classification 


Percentage  of  Time 
Devoted  to  Job 


Very  high  15  and  up 

High  6  -  14.99 

Moderate  2  -   5.99 

Low  0  -   1.99 


For  the  training  courses  being  evaluated,  the  three 
indexes  each  view  the  data  in  a  different  manner.   When  con- 
sidered collectively,  they  provide  a  thorough  indication  of 
the  extent  to  which  the  training  program  is  preparing  its 
graduates  for  performance  on  the  job.   No  one  index  can  be 
used  to  determine  absolutely  if  one  course  is  more  effective 
than  another,  nor  is  any  one  index  necessarily  the  one  by 
which  a  particular  program  should  be  evaluated.   Basically, 
all  three  indexes  must  be  considered  together  and  in  rela- 
tion to  one  another. 

Without  complete  knowledge  of  the  training  objectives 
and  the  administrative  constraints  associated  with  the 
training  program,  it  is  almost  impossible  to  assign  a  stan- 
dard which  the  indexes  should  have  met  in  order  for  the 


42 


courses  to  be  considered  effective.   However,  from  a  more 
academic  standpoint,  the  deviation  of  the  training  index 
from  1  and  the  undertraining  and  overtraining  indexes  from 
0  seem  to  be  rather  small  for  the  three  courses  being 
examined.   Therefore,  a  tendency  toward  an  effective  rating 
is  suggested,  although  an  effective-ineffective  judgement 
at  this  stage  of  the  analysis  is  somewhat  impractical. 

A  thorough  examination  of  each  task  located  off  the 
main  diagonal  in  the  classification  matrix  would  have  to 
be  conducted  in  order  to  determine  if  adjustments  in 
training  emphasis  would  be  feasible.   Nevertheless,  the 
training  indexes  as  presented  increase  the  understanding 
of  the  effects  of  the  training  program  and  provide  a 
quantitative  basis  for  altering  training  emphasis. 
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V.   A  PARAMETRIC  METHOD 

A  possible  parametric  procedure  for  determining  the 
quality  of  the  training  program  is  the  student's  "t"  test. 
The  usefulness  of  the  test  is  limited  to  an  effectiveness 
evaluation  on  each  individual  task  element.   Basically, 
it  allows  a  statistical  test  to  be  conducted  on  the  extent 
to  which  each  task  element  of  a  training  curriculum  is  job 
oriented.   From  the  calculation  of  test  statistics  for 
each  task  statement  and  the  subsequent  determination  of 
statistical  significance,  one  could  make  an  accurate 
analysis  of  the  emphasis  of  the  training  content.   Because 
of  the  large  number  of  calculations  involved,  it  is 
beyond  the  scope  of  this  paper  to  employ  the  parametric 
procedure  mentioned  above.   This  chapter  will  be  devoted 
to  an  explanation  of  the  procedure  and  the  assumptions  that 
are  required  for  using  the  "t"  test  in  a  meaningful  way. 

The  problem  becomes  one  of  testing  the  hypothesis  that 
the  mean  of  a  population  equals  a  specified  value  against 
the  alternative  that  the  population  mean  is  not  equal  to 
the  specified  value.   The  hypothesis  is  expressed  as 
follows : 

V   yi  =  yoi 

Hr    "i  '  ^oi 
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where 


where : 

y.  =  mean  Importance  level  of  task  1 

u  .  =  established  proficiency  level  associated  with 
task  i  in  the  training  curriculum. 

The  test  Is  based  on  the  reasonableness  of  the  assumption 

that  the  population  of  importance  levels  are  normally 

distributed  [2] .   If  this  assumption  is  feasible  within  a 

reasonable  degree  of  approximation,  then  the  test  statistic 

is  as  follows : 

X,  -  y  . 

t    = 
obs 


s//n 

X.  =  the  mean  of  the  sampled  population 

s   =  the  variance  of  the  sampled  population 

n   =  the  number  of  job  incumbents  from  which 
data  was  collected. 

The  critical  value  is  extracted  from  any  standard  "t"  table 

and  is  based  on  n-1  degrees  of  freedom  as  well  as  the 

desired  level  of  significance.   If  t  observed  is  greater 

than  t  critical,  the  null  hypothesis  is  rejected  and  it  may 

be  stated  that  training  is  not  job  oriented  for  the  task 

element  tested. 

This  same  procedure  is  then  repeated  for  each  element 
of  the  program  so  that  either  an  acceptance  or  rejection 
of  the  null  hypothesis  is  accomplished  for  every  task 
statement . 

The  t  test  used  in  this  manner  has  the  advantage  of 
evaluating  the  significance  of  every  element  of  the  program 
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and  provides  accurate  statistical  information  for  altering 
the  emphasis  of  course  contents.   Since  a  large  number  of 
task  elements  imply  extensive  calculations,  computer 
techndiques  are  suggested  for  computational  efficiency. 
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VI.   AN  OPERATIONAL  APPROACH  TOWARD  TRAINING  EVALUATIONS 

Previous  chapters  of  this  paper  discussed  nonparametric 
and  parametric  procedures  for  evaluating  the  effectiveness  of 
military  training.   Alternative  approaches  were  illustrated 
providing  statistical  techniques  for  analyzing  the  extent  to 
which  training  Is  job  oriented.   These  procedures  provide 
insight  into  the  emphasis  of  course  contents  based  upon 
proven  statistical  methods.   However,  it  still  remains 
impractical  to  assign  an  effective-ineffective  rating  to  any 
training  curriculum,  and  therefore,  evaluations  based  on  the 
operational  significance  of  the  data  are  also  required. 

As  a  possible  approach  toward  an  appraisal  of  the  train- 
ing program  in  a  more  operational  manner,  it  appears 
appropriate  to  consider  the  classification  procedure  demon- 
strated in  the  chapter  on  the  matrix  method  of  evaluation. 
The  statistical  procedures  demonstrated  in  Chapter  III  may 
provide  equally  acceptable  starting  points;  however,  only 
the  matrix  method  will  be  pursued  here.   Tasks   located 
off  the  main  diagonal  of  the  classification  matrix   are 
representative  of  candidates  for  a  more  intensive  examina- 
tion of  proper  training  emphasis.   Those  tasks  assigned  to 
cells  D,C,H,I,N,  and  M  of  the  matrix  should,  therefore,  be 
of  particular  concern  to  the  directors  of  the  training 
program. 

Subsequent  to  determining  candidate  tasks  for  a  more 
detailed  analysis  on  an  individual  basis,  it  seems 
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appropriate  to  examine  the  job  data  for  trends  or  patterns 
based  on  the  nature  of  the  selected  tasks.   Comparisons  of 
these  trends  or  patterns  associated  with  the  importance  of 
the  tasks  to  performance  on  the  job  against  those  related 
to  training  emphasis  provide  information  as  to  the  timeli- 
ness of  the  training  program.   More  specifically,  it  is  not 
only  important  to  determine  what  training  should  be  given, 
but  when  it  should  be  given  in  order  to  be  timely.   Dis- 
crepancies at  this  stage  of  the  analysis  indicate  an 
untimely  training  program  where  individuals  are  either  being 
trained  prematurely  or  subsequent  to  the  time  that  they  are 
required  to  perform  the  task. 

As  an  example  of  the  timeliness  concept,  consider  a 
complex  task  not  normally  required  of  lower  ranking  indi- 
viduals but  continuously  becoming  more  important  as  the 
individual  increases  in  rank.   A  timely  training  program 
would  not  provide  detailed  instruction  on  this  task  for 
the  entry  level  ranks  but  would  emphasize  the  task  within 
the  more  advanced  courses. 

Alterations  in  training  emphasis  indicated  by  an 
untimely  training  program  would  include  the  elimination  of 
instructional  areas  from  one  course  while  adding  them  to 
another.   In  keeping  with  the  objective  of  military  train- 
ing, these  additions  and  deletions  should  be  accomplished 
in  accordance  with  the  importance  of  the  tasks  to  be 
performed  on  the  job.   Other  tasks  receiving  timely  training 
may  still  require  modifications  in  emphasis  in  order  to 
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bring  the  program  more  on  target.   Adjustments  in  course 
contents,  when  the  timeliness  of  training  is  not  an  issue, 
require  an  examination  of  the  courses  on  an  individual 
basis  since  a  task  receiving  proper  emphasis  in  one  course 
may  be  overemphasized  or  underemphasized  in  another.   More 
specifically,  the  evaluation  procedure  suggested  here  is 
associated  with  determining  when  training  should  be  accomp- 
lished to  be  timely  as  well  as  determining  the  proper 
emphasis  that  should  be  devoted  to  specific  instructional 
areas  within  individual  courses. 

As  an  example  of  the  concepts  discussed  above,  consider 
task  Numbers  Three  and  Sixteen.   The  tasks  are  listed  on  the 
job  inventory  questionnaire  displayed  in  Appendix  C.   Task 
Number  Three  is  "the  assignment  of  work  to  personnel  on  a 
daily  basis."   The  data  also  indicates  that  the  frequency  of 
performance  of  this  task  is  continuously  increasing  as 
individuals  advance  in  rank.   Because  task  Number  Sixteen 
is  more  menial  in  nature,  its  associated  importance  level 
decreases  in  accordance  with  the  rank  structure. 

As  previously  mentioned,  the  job  inventory  data  was 
separated  into  three  distinct  categories  representative  of 
the  Marine's  most  recent  course  completion.   Table  VII 
illustrates  the  range  of  ranks  associated  with  each  category 
as  well  as  the  average  percentage  of  time  devoted  to  task 
Number  Three  on  the  job.   Table  VIII  shows  the  average  per- 
centage of  time  devoted  to  task  Number  Sixteen  within  the 
same  three  categories  of  data. 
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TABLE  VII 

SUMMARY  OF  JOB  INVENTORY  DATA  COLLECTED 
FOR  TASK  NUMBER  THREE 


Data  Category 

Rank 

10 

Job  Time 

Basic 

E2-E3 

2.14 

Intermediate 

E4-E6 

H.hi 

Senior 

E7-E8 

7.23 

An  observation  of  the  curriculum  contents  reveals  the 
same  continuous  tendencies  as  illustrated  in  Table  VII  and 
Table  VIII.   These  results  are  indicative  of  a  timely 
training  program  in  which  individuals  are  instructed  at 
the  proper  rank  for  adequate  job  performance. 

TABLE  VIII 

SUMMARY  OF  JOB  INVENTORY  DATA  COLLECTED 
FOR  TASK  NUMBER  SIXTEEN 

Data  Category        Rank         %   Job  Time 

Basic  E2-E3  8.5^ 

Intermediate         E4-E6  6.62 

Senior  E7-E8  3.31 

In  contrast,  task  Number  Eleven  as  listed  on  the  ques- 
tionnaire displayed  in  Appendix  C,  involves  the  maintenance 
of  financial  records.   The  data  shows  that  this  task  is 
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performed  more  regularly  by  lower  ranking  individuals  while 
training  emphasis  is  more  concentrated  toward  the  senior 
enlisted  man.   Task  Number  Eleven  is  therefore  representa- 
tive of  an  untimely  aspect  of  the  training  program  and 
should  receive  a  more  intensive  examination. 

Once  tasks  have  been  evaluated  for  timeliness  as  demon- 
strated above,  it  still  remains  to  examine  the  emphasis 
devoted  to  them  within  each  individual  training  course. 
This  portion  of  the  analysis  involves  the  modification  of 
course  contents  in  accordance  with  the  importance  of  tasks 
associated  to  performance  on  the  job  and  could  be  accomp- 
lished by  adjusting  training  emphasis  enough  to  relocate 
these  tasks  on  the  main  diagonal  of  the  classification 
matrix. 

However,  it  still  remains  to  consider  the  administra- 
tive type  constraints  associated  with  formal  military 
training  prior  to  the  adjustment  or  reorganization  of  course 
contents.   Since  these  constraints  are  obviously  varied  and 
unique  to  specific  training  programs,  they  will  not  be 
included  here  but  are  considered  an  appropriate  area  for 
future  research.   Although  a  specific  judgment  of  the 
effectiveness  of  the  program  remains  impractical,  a  more 
operational  approach  toward  the  employment  of  task  analysis 
in  the  evaluation  of  military  training  has  been  introduced. 
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VII.   A  GENERAL  METHODOLOGY 

This  chapter  introduces  a  general  systematic  methodology 
for  utilizing  task  analysis  in  the  evaluation  of  military 
training.   The  major  features  of  the  proposed  methodology 
have  emerged  from  modern  training  research  and  include 
statistical  techniques,  operational  evaluations,  and  compari- 
sons of  data  collection  methods.   The  procedure  consists  of 
ten  steps  to  training  curricula  evaluation: 

1.  Conduct  a  systems  analysis  of  the  program  to  be 
evaluated,  thereby  determining  the  rank  and  I^OS  requirements 
of  the  individuals  to  be  employed  as  subjects. 

2.  Determine  the  size  and  location  of  a  representative 
sample  of  job  incumbents  from  which  job  specification  data 
can  be  obtained  and  consider  sample  stratification  techniques 
if  infeasible  to  include  the  entire  population. 

3.  Develop  the  task  analysis  inventory  to  be  adminis- 
tered to  job  incumbents.   The  foundation  for  this  development 
comes  from  the  systems  analysis  of  the  training  program  as 
well  as  established  job  requirements. 

Jj .   Select  the  data  collection  method  to  be  used  and 
conduct  the  task  analysis. 

5.  Select  the  appropriate  statistical  procedure. 

6,  Consolidate  the  job  data  so  that  a  comparison  with 
the  training  proficiency  levels  can  be  conducted  in  accor- 
dance with  the  statistical  technique  selected. 
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7.  Conduct  the  statistical  test  and  determine  if  sta- 
tistical significance  exists. 

8.  Examine  available  data  from  an  operational  stand- 
point to  include  a  consideration  for  the  timeliness  of 
training. 

9.  Adjust  the  training  program  where  feasible. 

10.   Monitor  graduates  on  a  periodic  basis  in  order  to 
maintain  task  analysis  data  in  a  current  status. 

The  methodology  suggested  here  is  certainly  general  in 
nature  and  presents  no  revolutionary  ideas  In  the  search  for 
an  efficient  way  to  evaluate  training.   It  does,  however, 
provide  some  procedural  guidance  for  using  task  analysis  in 
the  evaluation  and  revision  of  military  training  curricula. 
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VIII.   VALIDATION  OF  DATA  COLLECTION  METHODS 

As  mentioned  in  Chapter  I,  a  secondary  objective  of  this 
paper  is  to  attempt,  on  a  relatively  small  scale,  statisti- 
cal comparisons  of  some  of  the  data  collection  methods 
associated  with  the  job  inventory  questionnaire.   This  par- 
ticular aspect  of  task  analysis  will  be  used  to  determine  if 
the  data  obtained  from  any  one  collection  method  is  signifi- 
cantly different  from  that  obtained  by  one  of  the  other 
procedures.   Chapter  I  also  lists  the  methods  of  data  col- 
lection most  feasible  for  Marine  Corps  task  analysis  efforts 
Three  of  these  four  methods  were  subjected  to  a  test  to 
determine  the  degree  of  association  existing  between  the 
three  distinct  combinations  of  data  gathering  procedures. 
The  task  analysis  data  collection  methods  selected  were  the 
most  feasible  for  this  type  of  comparison  and  are  as  follows: 

1.  Method  Number  1  -  Personal  interview  and  actual 
observation  of  an  individual's  on  the  job  performance.   The 
investigator  completed  the  questionnaire  based  upon  verbal 
responses  of  the  job  incumbents. 

2.  Method  Number  2  -  Distribution  of  the  task  analysis 
questionnaire  in  a  manner  not  permitting  observation  or 
interview  of  the  job  incumbent.   The  workers  completed  the 
questionnaire  without  assistance  from  a  trained  observer. 

3.  Method  Number  3  -  Distribution  of  the  questionnaire 
to  supervisors  reporting  on  the  job  specifications  required 
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of  their  subordinates.   Supervisors  completed  the  question- 
naire without  direct  assistance  from  a  trained  observer. 

A.   DATA  COLLECTION 

The  data  for  this  segment  of  the  paper  was  gathered  over 
a  period  of  about  six  weeks.   Fifteen  Marines  from  the 
Marine  Corps  Supply  Center,  Albany,  Georgia,  were  employed 
as  subjects.   The  Marines  held  a  rank  of  either  E-3  or  E-4. 

Initially,  staff  non-commissioned  officers  and  officers 
completed  the  inventory  questionnaire  on  the  job  character- 
istics that  they  felt  were  required  of  their  men.   The 
questionnaire  was  at  the  same  time  distributed  to  the  fifteen 
job  incumbents  in  accordance  with  data  collection  method 
Number  Two.   The  subjects  received  a  brief  explanation  of 
the  instructions  and  were  then  given  a  period  of  about 
twenty-four  hours  to  complete  the  questionnaire.   No  explan- 
ation of  the  individual  task  statements  was  permitted 
although  precautions  were  not  taken  against  subjects  dis- 
cussing some  of  the  statements  or  procedures  prior  to 
marking  the  questionnaire. 

Finally,  after  a  period  of  about  six  weeks,  the  subjects 
were  interviewed  on  the  job.   For  this  portion  of  the 
experiment  the  subject  considered  each  task  statement  on 
the  questionnaire  with  the  investigator  and  actually  was 
required  to  explain  why  he  felt  that  particular  tasks  were 
required  in  the  performance  of  his  job.   The  investigator 
then  marked  the  questionnaire  in  accordance  with  the  res- 
ponse of  the  job  incumbent. 
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The  same  questionnaire  was  employed  throughout  the  test 
and  is  shown  in  Appendix  C.   In  order  to  limit  the  selection 
of  responses  in  regard  to  the  extent  to  which  specific 
tasks  are  required  for  performance  on  the  job,  a  rating 
scale  containing  only  four  points  was  used.   An  explanation 
of  the  four  point  selection  scale  is  also  presented  in 
Appendix  C. 

Additionally,  a  questionnaire  useful  in  evaluating  the 
extent  to  which  human  factor  characteristics  or  worker 
characteristics  are  required  for  successful  performance  on 
the  job  was  administered  to  these  same  subjects  at  the  same 
time  as  the  performance  task  questionnaire.   This  particular 
questionnaire  was  explained  in  Chapter  I  and  was  developed 
by  Professor  Gary  K.  Poock,  a  member  of  the  Operations 
Analysis  faculty  at  the  Naval  Postgraduate  School.   This 
questionnaire  is  displayed  in  Appendix  B. 

B.   DATA  ANALYSIS 

Observation  of  the  completed  questionnaires  revealed  the 
development  of  an  apparently  significant  trend  in  regard  to 
the  three  data  collection  methods.   The  distribution  method, 
In  which  individual  workers  were  required  to  complete  the 
questionnaire  on  their  own,  without  detailed  explanations  of 
the  instructions,  yielded  higher  total  point  counts  in  all 
cases  when  compared  with  the  investigator  conducted  inter- 
view and  a  higher  total  point  count  in  ten  of  the  fifteen 
cases  when  compared  with  the  supervisors  reports. 
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Additionally,  the  results  of  the  questionnaire  when  used  by- 
supervisors  to  report  on  their  subordinates  (data  collection 
method  Number  Three)  demonstrated  a  higher  total  score  for 
fourteen  of  the  fifteen  subjects  when  compared  with  the  job 
interview  method  administered  by  the  investigator.   Total 
point  count  was  obtained  by  adding  the  responses  of  each 
subject  on  both  of  the  questionnaires  that  were  used  in  each 
phase  of  the  test. 

C.   TESTING  FOR  STATISTICAL  SIGNIFICANCE 

The  nonparametric  Sign  Test  was  used  to  test  for  statis- 
tical significance  between  the  pairs  of  data  gathering 
procedures.   Because  of  the  large  sample  size  of  task 
statements  the  normal  approximation  to  the  binomial  distri- 
bution was  required  for  computation  of  the  test  statistic. 
For  a  detailed  discussion  of  the  rationale  and  method  of 
this  test  see  [6].   Three  separate  test  statistics  were 
calculated  so  that  the  following  statistical  comparisons 
could  be  made: 

1.  Method  1  versus  Method  2 

2.  Method  1  versus  Method  3 

3.  Method  2  versus  Method  3 

The  following  null  hypothesis  and  alternative  hypothesis 

were  used  throughout  the  testing: 

Ho:   there  is  no  significant  difference  between  the 
data  gathering  methods. 

H]_:   there  is  a  significant  difference  between  the 
methods . 

The  results  of  the  tests  are  indicated  in  Table  IX. 
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TABLE  IX 

SUMMARY  OF  THE  SIGN  TEST  RESULTS  ON  THE 
PAIRS  OF  DATA  COLLECTION  METHODS 


Test  Critical  Value  P  Value 

1  vs  2             3.38  .0010 

1  vs  3             3.02  .0026 

2  vs  3             1.53  .1260 


Statistically,  the  results  of  this  test  are  determined 
by  comparing  the  P  value  with  the  desired  significance  level 
(a).   The  null  hypothesis  can  be  rejected  if  P  is  less  than 
or  equal  to  the  selected  significance  level.   As  an  example, 
if  a  =  .005  is  chosen,  the  results  indicate  a  significant 
difference  between  data  collection  method  Number  One  and  the 
other  two  procedures.   No  statistical  significance  is 
demonstrated  between  methods  Two  and  Three.   Statistically, 
the  results  indicate  that  job  data  obtained  by  distributing 
questionnaires  to  job  incumbents  is  not  different  from  that 
collected  by  supervisors  using  the  questionnaire  to  report 
on  the  job  specifications  required  cf  their  subordinates. 
Additionally,  the  results  of  the  Sign  Test  indicate  that 
the  data  obtained  by  personal  Interview  and  actual  observa- 
tion of  the  job  incumbents  is  different  from  that  obtained 
using  the  other  two  methods.   The  Sign  Test,  however,  does 
not  indicate  which  data  collection  method  produces  the  most 
valid  information 
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D.   ADDITIONAL  INFORMATION 

In  addition  to  the  statistical  results  of  the  Sign  Test, 
it  is  of  interest  to  consider  some  specific  items  on  the 
worker  characteristic  questionnaire  in  an  attempt  to  deter- 
mine which  of  the  tested  data  collection  methods  produce  the 
most  valid  Information.   Since  this  questionnaire  was 
designed  for  use  on  all  MOS  '  s  ,  some  questions  are  completely 
irrelevant  to  the  type  jobs  considered  for  this  evaluation. 
Of  particular  interest  is  Question  Twenty-nine  which  querries 
the  subject  as  to  his  need  for  an  accurate  sense  of  taste, 
and  Question  Fifty-five  which  asks  the  worker  if  the  ability 
to  cause  subordinates  to  willingly  produce  desired  results 
is  required  in  the  performance  of  his  job.   These  particular 
worker  characteristics  are  representative  of  those  definitely 
not  required  of  any  of  the  individuals  involved  in  the  test- 
ing.  None  of  the  job  incumbents  was  in  any  way  associated 
with  tasting  food  in  the  performance  of  his  job.   Addition- 
ally, none  of  the  subjects  had  the  responsibility  of  sub- 
ordinates nor  were  they  required  to  possess  the  capacity  to 
lead  or  direct  others  in  the  performance  of  their  jobs. 

In  this  regard,  it  appears  significant  that  eight  of  the 
fifteen  subjects  answered  question  twenty-nine  with  a  res- 
ponse greater  than  zero  when  the  questionnaire  was  adminis- 
tered via  distribution  and  without  detailed  verbal  instruc- 
tions.  Additionally,  fourteen  of  the  fifteen  subjects 
responded  with  a  score  greater  than  or  equal  to  two  on 
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question  fifty-five  when  this  same  data  collection  method 
was  employed.   Other  similar  inconsistencies  were  evident, 
revealing  that  data  obtained  via  distribution  of  the  task 
analysis  inventory  could  not  be  considered  valid.   Similar 
although  less  frequent  inconsistencies  appeared  when  the 
questionnaire  was  distributed  to  supervisors  who  employed 
it  to  report  on  the  owrker  characteristics  required  of  their 
subordinates . 

A  possible  explanation  of  these  results  might  be 
attributed  to  the  hesitancy  of  any  one  to  admit  that  very 
few  specific  characteristics  are  required  in  the  performance 
of  his  job.   Quite  naturally,  humans  associate  with  their 
jobs  a  complex  set  of  procedures  requiring  a  variety  of 
human  characteristics  as  well  as  a  vast  number  of  learned 
skills. 

Although  the  number  of  subjects  employed  was  small  and 
not  representative  of  the  entire  population,  the  personal 
interview  appears  to  be  the  superior  data  collection  method 
of  those  tested.   Regardless  of  the  explanations  or  conclu- 
sions involved,  this  portion  of  the  paper  represents  an 
area  of  task  analysis  research  that  could  and  should  be 
expanded. 
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IX.   SUMMARY  AND  CONCLUSIONS 

This  paper  has  investigated  some  of  the  problems  of 
present  day  training  evaluation  techniques,  and  has  been 
devoted  to  describing  methods  for  analyzing  training 
effectiveness  using  valid  task  analysis  inventory  data. 
Hopefully,  it  may  assist  in  bringing  training  emphasis  more 
on  target. 

Task  analysis  data  was  collected  by  the  investigator 
and  subjected  to  parametric  and  nonparametric  procedures 
for  statistically  determining  the  extent  to  which  training 
was  job  oriented.   Of  particular  interest  was  the  matrix 
method  of  evaluation,  which  appears  to  be  the  most  flexible 
procedure  based  on  the  nature  of  available  data. 

A  general  methodology  for  analyzing  training  programs 
using  task  analysis  was  presented.   Included  in  this  method- 
ology was  the  importance  of  operational  considerations  such 
as  the  timeliness  of  training.   In  this  regard,  an  evalua- 
tion approach  originating  with  the  classification  of  tasks 
by  the  matrix  evaluation  method  was  presented. 

Although  three  specific  supply  oriented  training  pro- 
grams were  subjected  to  the  analysis  techniques  presented, 
it  was  obviously  impractical  to  assign  an  effective  or 
ineffective  rating  to  any  one.   A  study  of  the  data  did, 
however,  suggest  the  necessity  for  some  changes  in  training 
emphasis  provided  the  administrative  and  cost  constraints 
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could  be  satisfied.   Of  particular  interest  was  that  the 
data  revealed  overtraining  as  significant  a  problem  as 
undertraining.   Although  cost  estimates  were  not  obtained, 
this  particular  aspect  of  training  analysis  could  and 
should  be  expanded  as  a  possible  approach  toward  signifi- 
cant training  cost  reductions. 

Additionally  three  distinct  questionnaire  oriented  data 
collection  procedures  were  subjected  to  a  small  scale 
experiment  to  determine  if  any  one  method  provided  the 
most  valid  information.   Although  this  experiment  was  based 
on  a  sample  size  of  only  fifteen  relatively  low  ranking 
enlisted  Marines,  it  was  concluded  that  data  can  best  be 
obtained  by  trained  observers  completing  the  proposed 
questionnaire  based  on  verbal  responses  from  the  job 
incumbents.   The  possibility  for  future  research  in  this 
area  was  also  indicated. 

Finally,  this  paper  has  attempted  to  indicate  the 
importance  of  improved  procedures  for  determining  the 
adequacy  of  training  content.   Curriculum  development  would 
certainly  profit  from  the  use  of  task  analysis  data  as 
feedback  to  determine  whether  training  programs  and  objec- 
tives satisfactorily  meet  job  specifications. 
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APPENDIX  A 

CURRICULUM  DESCRIPTIONS  AND  ASSOCIATED 
RANK  REQUIREMENTS 


BASIC  SUPPLY  COURSE:   This  course  is  designed  to  train 
Privates  (E-l)  through  Lance  Corporals  (E-3)  ,  who  have 
limited  or  no  supply  experience,  with  only  the  basic  funda- 
mentals of  the  technical  skills  required  of  supply  clerks. 
Basic  skills  taught  include  the  preparation  of  routine 
correspondence  and  reports,  use  of  supply  publications  and 
stock  lists,  computation  of  allowances ,  and  a  familiariza- 
tion of  basic  property  control  and  small  purchase  procedures. 

SUPPLY  ADMINISTRATIVE  COURSE:   This  course  is  designed  to 
train  Corporals  (E-4)  through  Staff  Sergeants  (E-6)  with 
some  of  the  formal  principles,  procedures  and  techniques 
related  to  the  supply  field.   The  course  provides  an  inter- 
mediate level  of  instruction.   Basic  tasks  taught  include 
the  following: 

1.  Additional  emphasis  on  the  tasks  included  in  the 
Basic  Course. 

2.  Supply  operational,  clerical,  and  administrative 
duties  incident  to  requisitioning,  procuring,  and  accounting 
for  supplies  and  equipment. 

3.  The  duties  of  a  stock  reviewer,  maintaining  card 
files,  as  well  as  screening  and  offsetting  transaction  cards 
with  appropriate  item  record  cards. 
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H ,      Mathematical  operations  on  the  electric  calculator. 

5.  The  general  operating  principles,  uses,  and  capabil- 
ities of  supporting  EAM/EDP  equipment. 

6.  Interpretation  of  computer  produced  reports. 

7.  Preparation  of  correspondence,  messages,  and  reports 
in  a  format  for  future  mechanization. 

SENIOR  COURSE :   This  course  is  designed  to  provide  senior 
staff  non-commisioned  officers  with  a  more  complete  under- 
standing of  the  entire  range  of  operational  principles  and 
methods  which  contribute  to  effective  management  within  the 
following  specific  task  elements 

1.  Supervising  the  duties  of  enlisted  personnel. 

2.  Coordinating  the  daily  supply  activities  of  a  unit 
or  organization. 

3.  Supervising  the  duties  of  personnel  involved  in 
purchasing  and  contracting  activities  of  a  unit  or  organi- 
zation . 

H.      Possessing  a  general  knowledge  of  procedures, 
directives  and  regulations  applicable  to  the  functional  areas 
of  the  supply  system. 

5.  Assisting  in  the  determination  of  supply  and  inven- 
tory management  programs . 

6.  Possessing  the  required  knowledge  for  the  adminis- 
tration and  operation  of  a  supply  activity  or  organization. 

7.  Having  an  understanding  of  basic  computerized  opera- 
tions and  their  application  to  supply  management  techniques. 
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APPENDIX  B 
HUMAN  FACTORS  IN  TASK  ANALYSIS 

A  number  of  the  human  characteristics  listed  below  may 
be  needed  in  the  satisfactory  performance  of  the  tasks 
included  in  your  job.   The  accompanying  form  is  designed  to 
gauge  your  estimation  of  the  necessity  of  the  character- 
istics, which  are  referred  to  as  Worker  Characteristics . 

In  this  section  devoted  to  Worker  Characteristics, 
indicate,  by  the  placement  of  an  "X"  in  the  appropriate 
column,  the  degree  of  each  characteristic  you  believe 
necessary  to  do  your  job.   The  definitions  of  the  several 
degrees  are  defined  as  follows: 

(0)  This  characteristic  is  not  required . 

(1)  This  characteristic  is  occasionally  required. 

(2)  This  characteristic  is  frequently  required,  but 
not  on  a  regular  repetitive  basis. 

(3)  This  characteristic  is  regularly  required,  the 
lack  of  which  renders  your  job  virtually  impossible 
to  perform. 


65 


WORKER  CHARACTERISTICS 


I.   PHYSICAL 

A.   STRENGTH:   In  the  four  categories  below, 
rate  only  muscular  strength,  not  muscu- 
lar endurance.   Considerations:   The  size 
weight  and  bulkiness  of  objects  handles; 
the  pressure  necessary  to  operate  con- 
trols and  tools. 

1 .  Finger,  hand,  wrist  and  forearm 
strength  —  That  necessary  to 
squeeze,  bend,  pull,  twist,  turn, 
shape  or  grip  objects. 

2.  Upper  arm  strength  —  That  necessary 
to  lift,  swing,  push,  pull,  carry  or 
throw  objects. 

3.  Back  and  shoulder  strength  —  That 
necessary  to  lift  objects  from  the 
floor,  move  objects  with  the  back 
and  shoulders  or  swing  heavy  tools 
to  strike  objects. 

4 .  Leg,  foot  and  ankle  strength  —  That 
necessary  to  lift  objects  using  knee 
action,  operate  pedals  with  pressure, 
grip  or  brace  with  the  knees  or  climb, 
kneel,  walk  or  stand  with  loads. 

B.   STAMINA :   In  the  following  six  categories, 

consider  both  muscular  endurance,  the  ability 
to  sustain  strength  over  a  period  of  time, 
and  circulo-respiratory  endurance,  that 
characteristic  commonly  referred  to  as  wind , 
the  ability  to  sustain  vigorous  activity  over 
a  period  of  time,  including  the  maintenance 
of  strength.   Consideration:  Pacing  by 
machines  or  superiors;  the  frequency,  dura- 
tion and  rapidity  of  movement;  the  extent 
of  vigorous  activity  such  as  lifting, 
climbing,  running,  crawling,  leaping,  etc.; 
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Level 

Required 

0  12  3 


Level 

Required 

0  12  3 


whether  the  duration  of  the  activity  is 
known  or  indeterminate  length. 

5 .  Rapid  work  for  a  series  of  short 
periods . 

6 .  Rapid  work  for  extended  periods. 

7 .  Rapid  work  for  indefinite  periods. 

8 .  Heavy  work  for  a  series  of  short 
periods . 

9.  Heavy  work  for  extended  periods. 
10.   Heavy  work  for  indefinite  periods. 

C.  DEXTERITY:   The  three  categories  immed- 
iately below  rate  the  skill  or  adroitness 
in  the  movement  of  the  subject  parts  of 
the  body.   Considerations:   the  speed, 
complexity,  and  repetitiveness  of  the 
movements;  the  accuracy  required; 
whether  or  not  all  digits  or  limbs  are 
used  to  complete  the  movement. 

11.  Finger  dexterity  —  Ability  to  move 
and  manipulate  objects. 

12.  Hand  and  arm  dexterity  (including 
the  fingers  as  part  of  the  whole). 
Ability  to  move  hands  and  arms 
accurately  and  quickly. 

13.  Foot  and  leg  dexterity  --  Ability 
to  move  feet  and  legs  accurately 
and  quickly. 

D.  COORDINATION:   Considerations  in  the  fol- 
lowing five  categories  include  the 
frequency,  complexity  and  repetitiveness 
of  movements . 

14.  Eye-hand  coordination  —  That  neces- 
sary to  control  the  hand  through  the 
use  of  vision. 
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15.  Foot-eye-hand  coordination  —  That 
necessary  to  control,  simultaneously, 
the  feet  and  hands  through  the  use 

of  vision. 

16 .  Coordination  of  the  independent 
movement  of  both  hands  —  That 
necessary  to  control,  simultaneously 
and  independently,  both  hands,  with- 
out necessarily  using  the  eyes.   Addi- 
tional considerations  are  the  distance 
the  hands  move  and  the  differences  in 
the  movement  of  the  two  hands.   Each 
hand  may  be  doing  something  different. 

17.  Foot-eye  coordination  —  That  neces- 
sary to  control  the  feet  through  the 
use  of  vision. 

18.  Foot-hand  coordination  —  That  neces- 
sary to  control  the  feet  and  hands 
simultaneously,  not  using  the  eyes. 

E.   SIZE  CONSIDERATIONS:  Any  boundaries  on 
your  physical  size,  upper  or  lower, 
necessitated  by  the  task  performed,  not 
by  regulations . 

19.  Height,  (fill  in  blank)  . 

20.  Weight,  (fill  in  blank)  . 


WORKING  CONDITIONS: 

21.   Unpleasant  conditions  —  To  work 

under  conditions  affecting  physical 
comfort.   These  include  bad  odors, 
excessive  noise,  vibration,  dust, 
dirt,  fumes  or  moisture,  high 
humidity,  extremes  in  temperature, 
constant  flux  or  a  wide  range  in 
temperature,  exposure  to  acids, 
unpleasant  sights. 
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22.  Hazardous  conditions  —  To  work  under 
conditions  affecting  physical  safety. 

II.  SENSORY  CHARACTERISTICS 

A.  VISUAL: 

23.  General  visual  acuity  —  That  neces- 
sary to  perceive  or  recognize 
objects,  locate  points  at  a  distance 
or  make  accurate  discriminations 
using  vision. 

24.  Color  discrimination  —  That  neces- 
sary to  distinguish  similarities  and 
differences  in  colors  and  shades 
thereof,  create  harmonious  combina- 
tions of  color  or  mix  or  match  colors. 

B.  VISUAL  -  MENTAL:   The  five  categories  be- 
low involve  judgments  requiring  the  use 
of  vision.   Considerations:  the  complexity 
of  the  objects  perceived;  the  frequency 
and  rapidity  of  the  required  observations; 
their  variability;  any  mechanical  aids 
used. 

25.  Size  estimation  —  That  necessary  to 
make  accurate  judgments  of  dimen- 
sions (height,  weight,  depth,  breadth 
or  thickness)  or  over-all  size,  area. 

26.  Quantity  estimation  —  That  necessary 
to  make  accurate  judgments  of  the 
number  or  capacity  of  objects. 

27.  Speed  estimation  —  That  necessary  to 
make  accurate  judgments  of  rate  of 
change  of  position,  involving  the 
estimation  of  both  time  and  distance. 
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28.  Quality  estimation  —  That  necessary 
to  judge  the  quality  or  workmanship 
of  material.   This  characteristic 
may  very  well  involve  the  use  of  the 
other  senses,  such  as  touch,  hearing, 
smell  or  taste. 

29.  Form  perception  —  That  necessary  to 
distinguish  correct  shape  or  outline 
or  generally  perceive  shape. 

C.  AUDITORY 

30.  General  keenness  of  hearing  —  That 
necessary  to  recognize  particular 
sounds  and  distinguish  similarities 
and  differences  in  pitch,  intensity 
and  quality  of  sounds. 

D.  OLFACTORY 

31.  Sense  of  smell  --  That  necessary  to 
recognize  particular  odors  or 
discriminate  differences  and 
similarities  in  the  quality  or 
intensity  of  odors. 

E.  TACTILE 

32.  Tough  discrimination  --  That  neces- 
sary to  accurately  judge  smoothness, 
roughness  and  other  surface  qualities, 
using  touch. 

33.  Muscular  discrimination  —  That 
necessary  to  make  judgments  based 
on  muscular  sensitivity,  such  as 
estimating  weight  or  resistance  by 
lifting,  pushing  or  pulling,  gauging 
position  or  guiding  body  members 
without  the  use  of  vision. 
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F.   TASTE 

3^.   Sense  of  taste  —  That  necessary  to 
accurately  distinguish  similarities 
and  differences  in  the  intensity  or 
quality  of  tastes,  or  recognize 
particular  tastes. 

III.   MENTAL  CHARACTERISTICS 

A.  MEMORY :   The  five  characteristics  below 
rate  the  capacity  of  the  mind  to  store 
images  for  future  references.   Con- 
siderations:  the  number  and  complexity 
of  images;  their  rate  of  acquisition; 
the  length  of  time  the  images  must  be 
remembered. 

35 .  Memory  for  concrete  details. 

36 .  Memory  for  ideas,  theories,  plans, 
processes,  policies. 

37.  Memory  for  oral  directions. 

38 .  Memory  for  written  directions. 
39 •   Memory  for  names  and  persons. 

B.  LEARNED  CHARACTERISTICS: 

*J0.   Arithmetic  computation  —  To  perform 
calculations  and  higher  mathematics. 

kl.      Planning  Ability  —  To  recognize  and 
comprehend  the  steps  necessary  to 
achieve  specific  ends,  decide  upon, 
set  up  and  coordinate  such  plans, 
and  organize  ideas  and  things. 

J|2.   Mechanical  Ability  —  To  understand 
and  use  the  principles  of  mechanical 
structure  and  operation  and  solve 
problems  involving  tools  and  machines. 

43.   Oral  Expression  —  To  express  oneself 
clearly  and  effectively,  in  use  of 
speech . 
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44.  Written  Expression  —  To  express 
oneself  clearly  and  effectively, 
presenting  information  and  ideas 
in  writing. 

C.   PSYCHOLOGICAL  CONSIDERATIONS 

45.  Attention  to  many  items  —  To  keep 
in  mind  many  parts  of  one  job, 
although  required  to  repeatedly  and 
constantly  shift  one's  attention 
from  part  to  part. 

46.  Adaptability  —  To  adjust  readily  to 
new  situations;  flexibility. 

47.  Decision-making  ability  —  To  be  able 
to  consider  evidence,  and,  with  a 
minimum  of  delay,  reach  a  reasoned 
conclusion . 

48.  Initiative  —  To  recognize  the  need 
for  a  change  in  procedures  and 
actions,  and  to  accomplish  such 
changes  without  specific  instructions. 

49.  Tact  —  To  use  diplomacy  in  dealing 
with  people  to  achieve  certain  ends 
(consider  only  situations  where  this 
characteristic  is  fairly  regular 
requirement  of  the  job). 

50.  Personal  appearance  —  To  maintain 
a  level  of  neatness,  grooming  and 
attire  above  and  beyond  that  con- 
sidered necessary  for  a  good  Marine. 

51.  Concentration  amid  distractions  — 
To  continue  performing  a  task  or 
tasks  amid  noise,  interruptions  or 
other  distracting  influences. 
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52.  Emotional  stability  —  To  maintain 
self-control  and  calmness  at  all 
times . 

53.  Dealing  with  the  public  —  To  meet 
and  deal  with  the  public,  not  neces- 
sarily involving  the  exercise  of 
tact,  but  maintaining  friendly 
relations . 

5^.   Teamwork  —  To  necessarily  subor- 
dinate one's  individual  performance 
to  the  good  of  the  team  or  unit. 

55.  Leadership  —  To  cause  subordinates 
to  willingly  produce  desired  results 
through  a  combination  of  superior 
knowledge,  thoughtfulness ,  courage 
and  exemplary  personal  performance. 

56.  Dependability  --  To  produce  desired 
results  at  or  prior  to  the  time  they 
are  needed. 

57.  Physical  courage  —  To  perform  the 
necessary  regardless  of  possible 
harmful  physical  consequences . 

58.  Moral  courage  —  To  do  what  is  right 
regardless  of  the  consequences. 

59.  Please  list  below  any  other  character- 
istics required  in  your  job  which 
have  not  been  covered  above.   Then 
check  the  level  of  the  characteristic 
required  in  your  job. 
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APPENDIX  C 
PERFORMANCE  TASK  QUESTIONNAIRE 

The  section  below  consists  of  a  list  of  actual  tasks 
possibly  included  in  your  military  occupational  specialty. 
Place  an  "X"  in  the  appropriate  column  to  indicate  the 
frequency  with  which  you  perform  each  task.   The  frequency 
is  defined  as  follows: 

(0)  I  do  not  perform  this  task. 

(1)  I  perform  this  task  occasionally. 

(2)  I  perform  this  task  frequently.   It  is  a  regular 
part  of  my  job. 

(3)  I  perform  this  task  virtually  to  the  exclusion  of 
others  included  in  my  MOS . 


TASK  ELEMENTS: 


PLANNING  AND  ORGANIZATION 


0  12  3 


1.  Organize  and  administer  the  supply  activi- 
ties for  an  FMF  unit,  post  or  station. 

2.  Plan  and  coordinate  the  daily  supply 
activities  of  FMF  unit,  post  or  station. 

3.  Assign  work  to  personnel  on  a  daily  basis. 

4.  Direct  the  training  of  supply  personnel. 

B.   ADMINISTRATION 

5.  Utilize  the  contents  of  basic  directives 
and  publications  pertaining  to  supply. 

6.  Utilize  publications  pertaining  to  sub- 
sistence accounting. 


7k 
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7.  Use  the  Marine  Corps  stack  lists  to  ver- 
ify stock  numbers,  nomenclature,  units 
of  issue  and  other  data  pertaining  to 
supply. 

APPLICATION 

8.  Prepare  or  edit  requisitions,  follow  ups 
and  cancellations  for  supplies  and  equip- 
ment . 

9.  Prepare  or  edit  invoices  and  vouchers 
pertaining  to  property  control  records. 

10.  Monitor  stock  levels  and  post  required 
entries  to  appropriate  accounts. 

11.  Maintain  financial  records. 

12.  Assist  in  the  preparation  of  a  budget. 

13.  Reconcile  fiscal  listings  and  assist  in 
the  management  of  funds. 

14.  Conduct  physical  inventories  of  supplies 
and  equipment . 

15.  Make  adjustments  to  property  records 
caused  by  investigations,  inventories  or 
other  property  accounting  transactions. 

16.  Operate  a  typewriter. 

17.  Operate  a  desk  calculator. 

18.  Establish  and  operate  a  routine  filing 
system. 

19.  Maintain  correspondence  and  directive 
files. 

20.  Employ  the  procedures  for  the  open  market 
purchase  of  supplies  and  equipment. 

21.  Maintain  an  allowance  list  for  supplies 
and  equipment . 

22.  Prepare  routine  correspondence  and  reports 
pertaining  to  supply. 
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23.  Maintain  informal  account  property 
records . 

24.  Compute  reorder  points  and  requisitioning 
objectives . 

25.  Establish  or  supervise  a  stock  control 
system. 

26.  Screen  and  offset  transaction  cards  with 
appropriate  item  record  cards. 

27.  Know  the  general  operating  principles, 
uses,  and  general  capabilities  of 
supporting  EAM/EDP  equipment. 

28.  Interpret  computer  and  EAM  produced 
reports  and  take  the  required  action. 

29.  Prepare  correspondence  or  reports  for 
key  punching. 

30.  Employ  the  principles  of  field  ware- 
housing to  include  the  proper  care  of 
material  in  storage. 

31.  Be  familiar  with  the  procedures  for  the 
shipment  of  Marine  Corps  material  to 
include  the  services  offered  and  the 
responsibilities  of  commercial  carriers. 

32.  Know  the  principles  and  procedures  for 
equipment  maintenance  utilized  by  the 
Marine  Corps. 
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APPENDIX  D 

CUMULATIVE  STEP  FUNCTION  DATA 
IN  KOLMOGOROV-SMIRNOV  TESTS 

BASIC  COURSE 


Task  No. 

Training  (CDF) 

Job  (CDF) 

d, 

1 

1 

.02 

.0018 

.0182 

2 

.02 

.0036 

.0164 

3 

.02 

.0250 

.0050 

4 

.02 

.0250 

.0050 

5 

.09 

.1015 

.0115 

6 

.10 

.1086 

.0086 

7 

.28 

.1816 

.0984 

8 

.40 

.2688 

.1312 

9 

.51 

.2955 

.2145 

10 

.56 

.3346 

.2254* 

11 

.56 

.3666 

.1934 

12 

.56 

.3755 

.1845 

13 

.56 

.3986 

.1614 

14 

.56 

.4591 

.1009 

15 

.57 

.5036 

.0664 

16 

.60 

.5890 

.0110 

17 

.60 

.6566 

.0566 

18 

.60 

.6993 

.0993 

19 

.61 

.7456 

.1356 

20 

.67 

.7634 

.0934 

21 

.68 

.77Q4 

.0994 

22 

.73 

.8257 

.0957 

23 

•  92 

.8542 

.0658 

24 

.96 

.8791 

.0809 

25 

.96 

.8791 

.0809 

26 

.96 

.8846 

.0754 

27 

.99 

.9273 

.0627 

28 

1.00 

.9842 

.0158 

29 

1.00 

1.00 

0 

30 

1.00 

1.00 

0 

31 

1.00 

1.00 

0 

32 

1.00 

1.00 

0 

*D  =  .2254 
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ADMINISTRATIVE  CURRICULUM 


Task  No. 

Training  (CDF) 

Job  (CDF) 

d. 

1 

1 

.025 

.0101 

.0149 

2 

.025 

.0149 

.0101 

3 

.025 

.0596 

.0346 

4 

.025 

.0719 

.0469 

5 

.080 

.0417 

.0617 

6 

.095 

.1453 

.0503 

7 

.215 

.2204 

.0054 

8 

.290 

.2830 

.0070 

9 

.330 

.3053 

.0247 

10 

.350 

.3482 

.0018 

11 

.400 

.3740 

.0260 

12 

.435 

.3863 

.0487 

13 

.445 

.4030 

.0420 

14 

.455 

.4210 

.0340 

15 

.465 

.4657 

.0007 

16 

.485 

.5319 

.0469 

17 

.510 

.5838 

.0738 

18 

•  515 

.6160 

.1010 

19 

.530 

.6625 

.1325 

20 

.560 

.6726 

.1126 

21 

.565 

.7002 

.1352 

22 

.595 

.7628 

.1678 

23 

.640 

.7968 

.1568 

24 

.645 

.8226 

.1776 

25 

.645 

.8274 

.1824* 

26 

.695 

.8375 

.1425 

27 

.790 

.8751 

.0851 

28 

.820 

.9306 

.1106 

29 

.840 

.9529 

.1129 

30 

.910 

.9761 

.0661 

31 

.970 

1.00 

.0300 

32 

1.00 

1.00 

0 

*D  =  .1824 
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SENIOR    COURSE 


Task  No. 

Training  (CDF) 

Job  (CDF) 

di 

1 

.0294 

.0351 

.0057 

2 

.0412 

.0652 

.0240 

3 

.0586 

.1364 

.0778 

4 

.0645 

.1886 

.1241 

5 

.1294 

.2538 

.1244 

6 

.1294 

.2628 

•  1334 

7 

.2533 

.3230 

.0697 

8 

.2827 

.3672 

.0845 

9 

.3063 

.3902 

.0839 

10 

.3122 

.4062 

.0940 

11 

.3712 

.4213 

.0501 

12 

.4125 

.4473 

.0348 

13 

.4243 

.4774 

.0531 

14 

.4243 

.5075 

.0832 

15 

.4479 

•  5325 

.0846 

16 

.4479 

.5646 

.1167 

17 

.4479 

.5997 

.1518 

18 

.4479 

.6379 

.1900 

19 

.4479 

.6790 

.2311 

20 

.4713 

.7111 

.2398 

21 

.4772 

.7412 

.2640 

22 

.5480 

.8014 

.2534 

23 

.5598 

.8274 

.2676 

24 

.5598 

.8334 

.2736* 

25 

.6129 

.8535 

.2406 

26 

.6129 

.8625 

.2496 

27 

.8666 

.9077 

.0411 

28 

.9138 

.9578 

.0440 

29 

.9492 

•  9749 

.0257 

30 

.9492 

.9799 

.0307 

31 

.9728 

.9900 

.0172 

32 

1.00 

1.00 

0.0 

*D  = 

=  .2736 
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APPENDIX  E 

TASK  TRAINING  AND  JOB  IMPORTANCE  RANKINGS  FOR  COMPUTATION 
OF  SPEARMAN  RANK  CORRELATION  COEFFICIENTS 

BASIC  COURSE 


Task  No. 

Trainin 

g  Rank 

Job  Rank 

1 

12 

25.5 

2 

25 

25.5 

3 

25 

18 

4 

25 

30 

5 

5 

3 

6 

15 

23 

7 

2 

4 

8 

3 

1 

9 

4 

16 

10 

7. 

5 

13 

11 

25 

14 

12 

25 

22 

13 

25 

18 

14 

25 

6 

15 

15 

10 

16 

10. 

5 

2 

17 

25 

5 

18 

25 

11.5 

.  19 

15 

8.5 

20 

6 

19.0 

21 

15 

20.5 

22 

7. 

5 

8.5 

23 

1 

15 

24 

Q 

" 

17 

25 

25 

30 

26 

25 

24 

27 

10. 

5 

11.5 

28 

15 

7 

29 

25 

20.5 

30 

25 

30 

31 

25 

30 

32 

25 

30 
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ADMINISTRATIVE  COURSE 


Task  No.        Training  Rank        Job  Rank 


27.0 
29.5 

9-5 
24.5 

2.0 
31.0 

1.0 

4.5 
19.5 
11.0 
16.5 
24.5 
23.0 
22.0 

9.5 

3.0 

7.0 
14.0 

8.0 
27.0 
15.0 

4.5 
13.0 
16.5 
29.5 
27.0 
12.0 

6.0 

19.5 
18.0 
21.0 
32.0 


1 

16.5 

2 

30.5 

3 

30.5 

4 

30.5 

5 

6.0 

6 

23.5 

7 

1.0 

8 

3.0 

9 

10.5 

10 

19.0 

11 

7.5 

12 

12.0 

13 

23.5 

14 

23.5 

15 

23.5 

16 

19.0 

17 

16.5 

18 

27.0 

19 

21.0 

20 

14.0 

21 

27.0 

22 

14.0 

23 

9.0 

24 

27.0 

25 

30.5 

26 

7.5 

27 

2.0 

28 

14.0 

29 

19.0 

30 

4.0 

31 

5.0 

32 

10.5 
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SENIOR  COURSE 


Task  No.        Training  Rank  Job  Rank 

1  12.5  11.5 

2  20.0  16.5 

3  17.5  1.0 

4  23.0  5.0 

5  4.0  2.0 

6  28.5  29.5 

7  2.0  3.5 

8  12.5  8.0 

9  15-0  22.0 

10  23.0  26.0 

11  6.0  27.0 

12  9.0  19.5 

13  20.0  16.5 

14  28.5  16.5 

15  15.0  21.0 

16  28.5  13.5 

17  28.5  H.5 

18  28.5  10.0 

19  28.5  9.0 

20  10.5  13.5 

21  23.0  16.5 

22  3.0  3.5 

23  20.0  19.5 

24  28.5  31.0 

25  7.0  23.0 

26  28.5  29.5 

27  1.0  7.0 

28  8.0  6.0 

29  10.5  25.0 

30  28.5  32.0 

31  15.0  28.0 

32  17.5  24.0 
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